Skip to content

Fix NoiseSensitivity ragas API compatibility#441

Closed
nibalizer wants to merge 1 commit intomainfrom
fix-noise-sensitivity-ragas-api
Closed

Fix NoiseSensitivity ragas API compatibility#441
nibalizer wants to merge 1 commit intomainfrom
fix-noise-sensitivity-ragas-api

Conversation

@nibalizer
Copy link
Contributor

@nibalizer nibalizer commented Oct 17, 2025

Pull Request Description

What and why?

Fixed a TypeError in the NoiseSensitivity test caused by incompatibility with the ragas library API. The ragas NoiseSensitivity metric no longer accepts a focus parameter in its constructor, which was causing the test to fail with TypeError: NoiseSensitivity.__init__() got an unexpected keyword argument focus.

Removed the focus=focus parameter from the noise_sensitivity() metric instantiation while keeping the parameter available in the wrapper function for backward compatibility.

How to test

Run the NoiseSensitivity test using the example code:

run_test(
    "validmind.model_validation.ragas.NoiseSensitivity",
    inputs={"dataset": vm_test_ds},
    params={
        "user_input_column": "question",
        "response_column": "rag_model_prediction",
        "reference_column": "ground_truth",
        "retrieved_contexts_column": "retrieval_model_prediction",
    },
).log()

The test should now execute without the TypeError.

What needs special review?

The focus parameter is still accepted by the wrapper function but is no longer passed to the underlying ragas metric. Verify that this doesn't break existing behavior or if the parameter should be removed entirely from the function signature.

Dependencies, breaking changes, and deployment notes

  • Requires ragas version >=0.2.3,<=0.2.7 (already specified in pyproject.toml)
  • No breaking changes to the public API

Release notes

Fixed compatibility issue with ragas library in NoiseSensitivity test that was causing TypeError when running the test.

Checklist

  • What and why
  • Screenshots or videos (Frontend)
  • How to test
  • What needs special review
  • Dependencies, breaking changes, and deployment notes
  • Labels applied
  • PR linked to Shortcut
  • Unit tests added (Backend)
  • Tested locally
  • Documentation updated (if required)
  • Environment variable additions/changes documented (if required)

@nibalizer @ampcode-com
Fix NoiseSensitivity ragas API compatibility
d6a12c5 
Remove unsupported 'focus' parameter from ragas NoiseSensitivity metric instantiation

Amp-Thread-ID: https://ampcode.com/threads/T-43be9938-fec7-4262-9667-1a8fa1aae3e3
Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: Amp <amp@sourcegraph.com>
@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2025

Pull requests must include at least one of the required labels: internal (no release notes required), highlight, enhancement, bug, deprecation, documentation. Except for internal, pull requests must also include a description in the release notes section.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2025

PR Summary

This PR removes the explicit 'focus' parameter from the call to the noise_sensitivity metric in the NoiseSensitivity test module. The change ensures that the metric is now invoked with its default configuration rather than using a specific focus. This update streamlines the metric invocation and aligns it with its revised parameter requirements as defined in the metric's implementation.

Test Suggestions

  • Add tests to verify that noise_sensitivity produces the expected output using its default configuration.
  • Test the functionality to ensure that removing the 'focus' parameter does not adversely affect overall behavior.
  • Include regression tests that compare previous and current outputs to validate the change.

@juanmleng
Copy link
Contributor

juanmleng commented Oct 21, 2025 •
edited
Loading

I am not able to reproduce this error. Using ragas 0.2.7

AnilSorathiya
AnilSorathiya reviewed Oct 22, 2025
View reviewed changes
Copy link
Contributor

@AnilSorathiya AnilSorathiya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not able to reproduce this either.

@juanmleng
Copy link
Contributor

juanmleng commented Oct 23, 2025

@nibalizer We can probably close this PR. We’ve capped the latest version of Ragas to 0.2.7, which implements the focus parameter. In newer versions—particularly version 0.2.14—the focus parameter was renamed to mode.

Ragas 0.2.7 (our version)

class NoiseSensitivity(MetricWithLLM, SingleTurnMetric):
    name: str = "noise_sensitivity"
    focus: t.Literal["relevant", "irrelevant"] = "relevant"
    _required_columns: t.Dict[MetricType, t.Set[str]] = field(
        default_factory=lambda: {
            MetricType.SINGLE_TURN: {
                "user_input",
                "response",
                "reference",
                "retrieved_contexts",
            }
        }
    )

Ragas 0.2.14 (when focus was renamed to mode)

class NoiseSensitivity(MetricWithLLM, SingleTurnMetric):
    name: str = "noise_sensitivity"
    mode: t.Literal["relevant", "irrelevant"] = "relevant"
    _required_columns: t.Dict[MetricType, t.Set[str]] = field(
        default_factory=lambda: {
            MetricType.SINGLE_TURN: {
                "user_input",
                "response",
                "reference",
                "retrieved_contexts",
            }
        }
    )

@nibalizer nibalizer closed this Oct 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AnilSorathiya AnilSorathiya AnilSorathiya left review comments

@juanmleng juanmleng Awaiting requested review from juanmleng

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nibalizer @juanmleng @AnilSorathiya